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In models of dynamical electroweak symmetry breaking due to strongly coupled fourth-family 
quarks and leptons, their low-energy effective descriptions may involve multiple composite Higgs 
fields, leading to a possibility that the electroweak phase transition at finite temperature is first 
order due to the Coleman- Weinberg mechanism. We examine the behavior of the electroweak phase 
transition based on the effective renormalizable Yukawa theory which consists of the fourth-family 
quarks and two SU(2)-doublet Higgs fields corresponding to the bilinear operators of the fourth- 
family quarks with/without imposing the compositeness condition. The strength of the first-order 
phase transition is estimated by using the finite-temperature effective potential at one-loop with 
the ring- improvement. In the Yukawa theory without the compositeness condition, it is found 
that there is a parameter region where the first-order phase transition is strong enough for the 
electroweak baryogenesis with the experimentally acceptable Higgs boson and fourth-family quark 
masses. On the other hand, when the compositeness condition is imposed, the phase transition 
turns out to be weakly first order, or possibly second order, although the result is rather sensitive 
to the details of the compositeness condition. Combining with the result of the Yukawa theory 
without the compositeness condition, it is argued that with the fourth-family quark masses in the 
range of 330-480 GeV, corresponding to the compositeness scale in the range of 1.0-2.3 TeV, the 
four-fermion interaction among the fourth-family quarks does not lead to the strongly first-order 
electroweak phase transition. 

PACS numbers: ll.10.Wx, 12.60.Fr, 98.80.Cq. 
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I. INTRODUCTION 

The standard model (SM) can in principle fulfill all 
three Sakharov conditions 111 for generating a baryon 
asymmetry in the universe The model fails, how- 

ever, for two reasons, to explain the value of the asym- 
metry required for the primordial nucleosynthesis [H, or 
the value measured through the cosmic microwave back- 
ground [6| . The first reason is that the CP violation from 
the Kobayashi-Maskawa mechanism 0] , which nicely ex- 
plains CP violation in K- and B-systems, is highly sup- 
pressed HEQIllll. The second reason is that 
the electroweak phase transition (EWPT) is not strongly 
first order. The experimental lower bound on the Higgs 
mass, m h > 114 Ge V fl^| , im plies that there is no EWPT 
in the SM [H, [H, O El- Consequently, sphaleron- 
induced (B+L)-violating interactions are not sufficiently 
suppressed in the broken phase and wash out the baryon 
asymmetry. Therefore, if the physics at the electroweak 
scale is to explain the baryon asymmetry in the universe, 
the better understanding of the structure of the Higgs 
sector and of the source of CP violation would be re- 
quired. 

The fourth family is still a viable phenomenological 
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possibility beyond the SM. The constraint from the in- 
visible Z width is insignificant for the fourth-family neu- 
trino being heavier than Though the electroweak 
precision data I19I I giv e stringent constraint on the fourth 
family [H l2ll |22L [2§j , it is known that the data do not 
exclude their existence [U, [2^, [26|, [27} • It was shown 
that there still remain a parameter region being consis- 
tent with all current experimental bounds [281]. 

The existence of the fourth-family quarks accommo- 
date the extra mixings and CP violating phases within 
the Cabibbo-Kobayashi-Maskawa scheme. It has been 
argued that the observed anomaly in B-CP asymmetries 
may be explained by the effect of the fourth- family quarks 
[29L f30l. [31L I32I] . Recently, it was pointed out that CP vio- 
lation from these new phases could be large enough to ex- 
plain the baryon asymmetry in the universe on the basis 
of the dimensional analysis using the Jarlskog invariants 
extended to four families [331] . 

The question is then whether the EWPT can be 
strongly first order with the fourth family: the mere ad- 
dition of the fourth family to the SM is of no help in 
this respect, as long as the standard Higgs sector with 
the single SU(2) doublet is considered. (See, for exam- 
ple, a recent study by Fok and Kribs j34[.) Carena et 
al. [HHl has first discussed the possibility of a first-order 
EWPT due to new heavy fermions coupled strongly to 
Higgs bosons. They found that some heavy and strongly- 
interacting bosonic fields are required both to stabilize 
the effective potential against the large effect of the heavy 
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fermions and to cause a first-order EWPT [l30j | . This led 
the authors to consider a supersymmetric model. The 
EWPT in the supersymmetric model with the fourth 
family has recently been examined in [34[. (See [36| for 
earlier works.) 

If the masses of the fourth-family quarks and lep- 
tons are quite large and are comparable to the uni- 
tarity bounds, the fourth family must couple strongly 
to the Higgs sector [33] ■ In this case, the masses of 
the fourth-family quarks and leptons (or their vacuum 
condensates) may be regarded as the order param eters 
of the electroweak symmetry breaking (EWSB) [l3l| 
[H, EE EE EE El m, H3- The effective description 
of the fluctuations of the order parameters may involve 
multiple Higgs scalar fields. This leads to the possibility 
that the EWPT would be first order due to the Coleman- 
Weinberg mechanism (th e fluctuation induced first-order 
phase transition) [Eil^IillllllEliEmSlii] flsl ]. 

The goal of this paper is to explore the above possibil- 
ity of a first-order EWPT due to the heavy fourth family. 
We start from a model of dynamical electroweak symme- 
try breaking due to the effective four-fermion interac- 
tions of the fourth-family quarks and leptons at the scale 
A4f around a few TeV. We adopt the four-fermion in- 
teractions considered by Holdom [72| . This four-fermion 
theory may be rewritten into a Yukawa theory by intro- 
ducing auxiliary scalar fields which corresponds to the 
bilinear operators of the fourth-family quarks and lep- 
tons. These scalar fields consist of three SU(2) doublets 
and one SU(2) triplet. (It is assumed that the right- 
handed neutrino is extra heavy, acquiring its mass at 
the flavor scale around 1000 TeV. ) The renormaliza- 
tion group evolution from the scale A4f down to the elec- 
troweak scale v (~ 246 GeV) may generate operators such 
as the kinetic and interaction terms of the scalar fields 
and other higher dimensional operators. We then extend 
this model by including the kinetic, cubic and quartic 
terms of the scalar fields so that it becomes renormaliz- 
able, neglecting the effect of the higher dimensional oper- 
ators. It is this effective renormalizable model for which 
we examine EWPT through the finite-temperature ef- 
fective pot ential at one-loop with the ring-improvement 
P, M IzE Izl M M HE HI . Strictly speaking, 

in our case, the renormalization group equations must 
be subject to the compositeness condition as a boundary 
condition at the scale \x = [3, HH . Accordingly, the 
values of the renormalized couplings at the lower scale 
fi = v are restricted in a certain region of the parameter 
space. In our analysis, however, we will first explore the 
parameter space of the renormalizable theory without the 
constraint due to the compositeness condition, in order to 
locate the parameter region where a strongly first-order 
EWPT is realized. We then examine the possible overlap 
of these two regions. 

In this paper (I), we concentrate on the effect of the 
heavy quarks and consider two SU(2) doublets out of 
four scalar fields. The bosonic sector of our model then 
reduces to the two Higgs doublet model (2HDM) [HUE 



H, HE [H, [H, M, M, M, M, H S . The analysis of 
the effect of the heavy charged lepton and neutrino will 
be reported in a subsequent paper [13] • We also neglect, 
in this paper, the SU(3)xSU(2)xU(l) gauge interaction 
and consider the global symmetry limit because we do 
not expect a large effect of the electroweak interaction to 
the dynamics of the first-order phase transition in this 
model |65j . 

This paper is organized as follows. In section [H] we 
formulate the effective Yukawa theory and introduce the 
cutoff scale A by considering the vacuum instability and 
the triviality bounds. We then specify the composite- 
ness condition for our model. In section IIII1 we derive 
the finite-temperature effective potential at one-loop with 
the ring- improvement. In section HVl based on the nu- 
merical analysis of the effective potential, we examine the 
strength of the first-order phase transition in the Yukawa 
theory with/without the compositeness condition. Sec- 
tion [V] is devoted to conclusion and discussion. 

II. FOURTH FAMILY AND ELECTROWEAK 
SYMMETRY BREAKING 

A. Fourth family and Four-fermion interactions 

We assume the existence of the fourth-family quarks 
and leptons, which we denote by q' = (t',b') T , £' L ~ 
{v' tL ,t' l ) t , t r . The right-handed neutrino v' rR is as- 
sumed to acquire its mass at the flavor scale around 1000 
TeV and to be absent below the flavor scale. To be con- 
sistent with the electroweak precision date, the masses 
of the fourth-family quarks should be almost degenerate 
with a small mass splitting. For simplicity, we assume 
m t , = m h , . 

Following Holdom [72j, we introduce the four-fermion 
interactions of the fourth- family fermions as follows: 

Cm =G q >(q' Li q Rj )(q' Rj q' Ll ) + G T > ut' r ){t' R l' Li ) 

-G <L {&e Lj )[i' Lj ci' T Li ), (i) 

where C is the charge conjugation matrix and color in- 
dexes are contracted within a bracket. The scale of 
these interactions is assumed to be A4f : G q ' , G T ' , G v > l ~ 
1/A| f . The interaction term among the quarks 
has SU(2) L xSU(2) i? xU(l)yxU(l) j4 symmetry, where 
U(l)y corresponds to the baryon number. On the 
other hand, the interaction terms among the leptons 
have SU(2)l xU(1)l xU(l)i? symmetry which includes 
the vector like U(l) symmetry corresponding to the lep- 
ton number. The above four-fermion interactions, there- 
fore, have the extra symmetries compared with the SM 
Higgs sector which has 0(4) symmetry. We then as- 
sume the existence of sub-leading multi-fermion opera- 
tors, which are suppressed compared with Eq. ([I]) , so that 
the extra symmetries are broken explicitly and, hence, 
the possible pseudo Nambu-Goldstone (NG) bosons ac- 
quire non-zero masses. 
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The four-fermion interactions may be rewritten into 
the form of Yukawa interactions by introducing the aux- 
iliary scalar fields <E>, H T i and x a i a = 1)2,3), which 
correspond to the bilinear operators of the fourth-family 
quarks and leptons as follows: 

~ Q'RjJu , H T , ~ r> R £' Li , X a ~ £' L T a eCi' T L . (2) 
Then one obtains 
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-m| tr(4+*) - m 2 H lQ U\,H T , - m 2 xoX a * X a + C Y , 

(3) 



where Cy is the Yukawa interaction terms given by 

C Y = - y q 'o(q' l^Qr + c - c ) - Vt>{£' lH t >t' r + c.c.) 

-f(e?CUT a X a £i L + c.c.). (4) 

B. Effective Renormalizable Theory 

Through the renormalization group evolution from the 
scale [i = A4f down to the electroweak scale fi = v 
(=246 GeV), the kinetic and interaction terms of the 
scalar helds and other higher dimensional operators may 
be generated. We then extend this model by including 
the kinetic, cubic and quartic terms of the scalar fields 
so that it becomes renormalizable, neglecting the effect of 
the higher dimensional operators. The effective renormal- 
izable theory is then given by the following Lagrangian: 



C = U + C Y - V, 



(5) 



where £k consists of the kinetic terms for fourth-family 
fermions and scalar bosons and V is the scalar pote ntial. 
The explicit form of V is given in appendix [A] [1331 ] . 

Strictly speaking, the renormalization group equations 
are subject to the compositeness condition as a bound- 
ary condition at the scale fi = A4f j44|. Accordingly, the 
values of the renormalized couplings at the lower scale 
fj, = v are restricted in a certain region of the param- 
eter space. In the following analysis, however, we will 
first explore the parameter space of the renormalizable 
theory without the constraint due to the compositeness 
condition, in order to locate the parameter region where a 
strongly first-order EWPT is realized. We then examine 
the possible overlap of these two regions. 

In this paper (I), we concentrate on the effect of the 
fourth-family quarks and consider two SU(2) doublets 
out of four scalar fields. We also neglect, in this paper, 
the SU(3)xSU(2)xU(l) gauge interaction and consider 
the global symmetry limit, simply because we do not ex- 
pect a large effect of the color and the electroweak inter- 
actions on the dynamics of the first-order phase transition 
in this model. Then the Lagrangian Eq. §5§ reduces to 

C = q'lfiq 1 -y{q' L <S>q' R + c.c.) 
+ tt(d^^) -m|tr$t$ 
Ai 



(tr$ f $) 



tr($ f $) 2 + c(det $ + c.c). (6) 



We include the last term which breaks the U(1)a sym- 
metry and induces the mass of the pseudo NG boson. 
Then, the symmetry of the theory is the chiral symmetry 
SU(2)£xSU(2)fl; plus the U(l)y symmetry correspond- 
ing to the baryon number. We do not include the terms 
which consist of e$*e other than in the determinant term. 



C. Electroweak Symmetry Breaking 

We assume that the chiral symmetry SU(2)ixSU(2)fl' 
breaks down to the diagonal subgroup SU(2)y by the 
vacuum expectation value (VEV) of 



(#) = 



-I, 



f 



(7) 



where Nt(= 2) is the number of the fourth-family quark 
flavors, I is the Nf x Nf unit matrix and <j> > 0. At tree- 
level, the VEV is determined by the effective potential: 



1 



+ i|A 



For (m| — c) < 0, the VEV is given by 



-2(m| - c) 
Ai + X 2 /N f ' 



(8) 



(9) 



For the effective potential to be stable in this channel, 
the following conditions must be satisfied: 



Ai + X 2 /N f > 0, A 2 > 0. 



(10) 



Around the VEV, we may parametrize the fluctuation 
of Q(x) as follows: 



a=l 



where a a (a = 1, 2, 3) are the Pauli matrices. The fields 
h, rj, £ Q , 7r Q and q' acquire masses at tree level as summa- 
rized in Table HI where, for notational simplicity, we use 
the following abbreviations: 

o/i = l^i+^/N f ), a ? = i(A 1 + 3A 2 /JV / ), (12) 
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a n = -(X 1 +X 2 /N f ), 



1 



2AT. 



■y 



and 



bh = a h - a n = (Ai + X 2 /N f ), 
= a,£ - a n = (X 2 /Nf). 



(13) 



(14) 
(15) 



The bosonic sector of this model is just the 2HDM. h is 
the singlet of SU(2)y and corresponds to the SM Higgs 
boson. The adjoint n a are the NG bosons of the break- 
ing of SU(2) i xSU(2) fi; , while the singlet r) is the pseu- 
doscalar Higgs boson and is also the pseudo NG boson 
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TABLE I: The effective masses and the numbers of the degrees 
of freedom in the model for the fourth family quarks 



associated with the breaking of the U(1)a symmetry. 
The adjoint £ Q consist of the extra neutral Higgs bo- 
son and the charged Higgs bosons. Three NG bosons 
7r Q are eaten by W and Z bosons when the electroweak 
interactions are introduced. As for the fourth-family 
quarks, the experimen tal l ower bound from the direct 
search m q i > 256 GeV 109| implies y > 2.1 at tree-level 
by taking <j) = v(= 246GeV). 



D. Cutoff Scale of the Effective Theory 



The applicability of the effective theory defined by the 
Lagrangian Eq. ([6]) would break down at some energy 
scale and one needs to introduce a cutoff A. The running 
coupling constants in this model, Ai(/i), A 2 (/i) and y(fi), 
obey the following renormalization group equations at 




200 500 1000 2000 5000 1 x 10 4 

rfGeV] 

FIG. 1: The renormalization group flows of the Yukawa cou- 
pling for various initial values at the electroweak scale. The 
values of y are 2.0, 2.5, 3.0 from bottom to top. The gray solid 
line indicates the value y — 2.1, which corresponds to the ex- 
perimental lower bound from the direct search of the fourth- 
family quarks, m q i > 256 GeV. The blue dashed line indi- 
cates the upper limit of the perturbativity condition, which 
is adopted in our analysis as a criterion for the Landau pole. 



one-loop: 

M^A 2 = ^2 (6AiA 2 + 2N f Xj + 2N c y 2 \ 2 - 2N c f), 



d 



1 



8/1 16-7T 



■(N f +N c )f 



(16) 



with the initial conditions Ai(i>) = Ai, ^2(v) = A 2 and 
y{v) = y given at the electroweak scale. As one can see in 
FIG. [l] the Yukawa coupling, which is large at the elec- 
troweak scale for the heavy fourth- family quarks, would 
blow up to infinity (due to the Landau pole) at a cer- 
tain energy scale not far from the electroweak scale [l34| . 
The ultraviolet behaviors of the scalar quartic couplings 
then take two types of pattern depending on the relative 
size of the scalar quartic couplings and Yukawa coupling 
at the electroweak scale: (i) the scalar quartic coupling 
Ai(/z) + \2{n)/Nf and/or A2(/x) are driven to negative at 
some energy scale, implying that the electroweak vacuum 
is unstable; (ii) the scalar quartic couplings encounter 
the Landau pole at some energy scale. In both cases, one 
should introduce a cutoff before these problems happen. 

We estimate the cutoff A, for given initial values of the 
couplings at the electroweak scale, as the scale at which 
one of the following conditions is first hit: 



Ai(A) + A 2 (A)/7V / = 0, A 2 (A) = 0, 



(17) 



which correspond to the case (i) (the vacuum instability) 
and 



y(A) 2 



16^2 



A 1 (A) + A a (A)/JV> = ^ J A 2 (A) 



16tH 



N 



f 



(18) 



(19) 



which correspond to the case (ii) (the Landau pole). 
Here, we adopt the upper limits of the perturbativity 
bounds, 



y(A) 2 < 



16^ 2 



Ai(A) + A 2 (A)/JV/ < 



16tt 2 



(20) 

A 2 (A)< 1^,(21) 



as a criterion for the Landau pole. 

In FIG. [H we show the contours of the estimated cut- 
off A for y = 2.0 in Ai-A 2 /7V/ plane. We see that in the 
most region, the cutoff scale is around 1 TeV or lower. 
For the fixed Ai, A tends to increase with A 2 /A/ for small 
A 2 / Nf and tends to decrease with A 2 /Nf for large A 2 /Nf . 
The former (latter) behavior is due to the fact that A 
is determined via the vacuum instability (the Landau 
pole) conditions in that region. FIG. [3] is the similar 
plot for y = 2.5. We see that for the larger value of the 
Yukawa coupling, the relatively larger values of A 2 /A/y 
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FIG. 2: An estimate of the cutoff scale A for y = 2.0 in 
A1-A2/-/V/ plane. The dashed, dot-dashed, solid and dotted 
contours correspond to A = 5.0, 1.5, 1.0, 0.5 TeV, respectively. 
In the shaded region, the effective potential at tree-level is 
unstable and we consider the region where Ai + A2/A'/ > 
only. 



y=2.5 




FIG. 3: An estimate of the cutoff scale A for y = 2.5 in 
X1—X2 /Nf plane. The dot-dashed, solid and dotted contours 
correspond to A = 1.5, 1.0, 0.5 TeV, respectively. 



are required to fulfill the vacuum stability condition. In 
FIG. |3J we show the contours of the estimated cutoff A 
for Ai + X 2 /N f = 0.05 in y-\ 2 /N f plane. 

In order to ensure the applicability of the effective 
renormalizable theory, the cutoff A should be taken to 
be large enough compared with other mass scales in the 
theory: A » m.;(0o),0o. In the following analysis of 
the first-order EWPT, it turns out that the largest mass 
scale is given by mj(0o) around 400-700 GeV. Then, 
we require A > 1 TeV and exclude the region of the 
parameter space where this condition is not fulfilled. 



FIG. 4: An estimate of the cutoff A for Ai + \ 2 /N f = 0.05 in 
y-\2/Nf plane. The dashed, solid and dotted contours corre- 
spond to A = 5.0, 1.0, 0.5 TeV, respectively. The blue filled- 
circle (indicated with "A"), which is located at the "cusp" 
on the boundary of the allowed region with A > 1.0 TeV, 
indicates the electroweak-scale values of the running coupling 
constants subject to the compositeness condition (Criterion 
A) at A 4 f = 1.0 TeV. 



This requirement leads to the constraint on the quark 
masses, m q ' < 370 GeV at tree-level, corresponding to 
the Yukawa coupling y < 3.0. 



E. Compositeness condition 

Just below the scale of the four-fermion interaction 
/i < A 4 f , the four-fermion theory Eq. (|3|) with only quark 
fields q' , 



C' 4{ = q'ilf 



y q 'o(q' l^Qr + cc) - m| trfW), (22) 



is renormalized to the Yukawa theory Eq. © , where the 
renormalized couplings are given by 



Ai(m) 
A 2 (/i) 



0, 

32^ 2 



1 



2/0) 



16tt 2 1 



(See appendix |B 
finds 

Ai(/i) 
A 2 (M) 

m 2 



N c ln(Ay M 2)- 
for detail.) In the limit /x 



0, 

00, 
00, 



(23) 
(24) 

(25) 
A 4 f, one 

(26) 



a 2 (m)Mm) 5 



This provides the compositeness condition in terms of the 
renormalized couplings as the boundary condition of the 
renormalization group equations at /1 = A 4 f [44j . 
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Criterion: A(dashed), B(dotted), C(dot-dashed) 




Criterion: A(dashed), B(dotted), C(dot-dashed) 
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from top left to bottom right, respectively. The case with 
A4f = 1.0 TeV comes close to the stability boundary at 
the electroweak scale, taking the value Ai+A 2 /V/ = 0.05. 
Then, in FIG.[U the values y, A 2 of this case are indicated 
by the blue filled-circle, which is located at the "cusp" 
on the boundary of the allowed region with A > 1.0 TeV. 

In fact, the electroweak-scale values of the couplings 
are rather sensitive to the choice of their values at the 
compositeness scale A4f. To see this, let us refer the 
above criterion for the compositeness condition as " A" , 
and introduce slightly modified criteria "B" and "C" as 
follows: 



Ai(A 4f ) = 0, A 2 (A 4f ) = — , y{k i{ ) z = — , 



A 2 (A 4f )/y(A 4f ) 2 =N C /N f 



and 



Ai(A 4f ) = 0, A 2 (A 4f ) 



4tt2 



A 2 (A 4f )/y(A 



Nc/N f 



[Criterion B], (28) 



[Criterion C]. (29) 



In FIG. O the red filled-circles and green filled-circles 
shows the electroweak-scale values X±, A 2 of the running 
coupling constants subject to the compositeness condi- 
tions Eqs. (|28|) and ([29]) . respectively. We will discuss 
this point further in section ITVl in relation to the analysis 
of EWPT. 



FIG. 5: The electroweak-scale values of the running coupling 
constants, which are subject to the compositeness conditions 
at various A 4 f, are shown in A1-A2/-/V/ plane. The blue, red, 
green data sets correspond to the criterion A, B, C, respec- 
tively. The blue filled-circles indicate the values Ai, A2/V/ for 
A 4f (y) equal to 0.5 TeV (4.0), 1.0 TeV (3.0), 1.5 TeV (2.7), 
5.0 TeV (2.2) from top left to bottom right, respectively. 



In our effective theory formulated as above, however, 
the compositeness condition should be modified. One can 
not impose the above condition Eq. ([2^]) literally because 
the values of the couplings A 2 (/i) and y(p) must exceed 
the perturbativity bounds. But, the divergence of A 2 (/Lt) 
and y(fJ>) is due to the Landau pole. Then, it seems 
reasonable in our case to substitute the upper limit of the 
perturbativity bounds for the compositeness condition: 

T IK \ n x IK \ 1671-2 —IK \2 1671-2 

Ai(A 4f ) = 0, A 2 (A 4f ) = — — , y(A 4l ) 2 = — — , 

j>if i\ c 

A 2 (A 4f )/y(A 4f ) 2 = N c /N f [Criterion A]. (27) 

In FIG.O we plot the electroweak-scale values Ai, A 2 of 
the running coupling constants which are subject to the 
compositeness conditions Eqs. (|2T)l at various scales. The 
blue filled-circles correspond to the values of A 4f (y) = 
0.5 TeV (4.0), 1.0 TeV (3.0), 1.5 TeV (2.7), 5.0 TeV (2.2) 



III. EFFECTIVE POTENTIAL 
A. Zero-temperature effective potential 

At zero temperature, the one-loop effective potential 
is given by 



v<°\4>) =v (<t>) + vi°\<f,) 



(30) 



where Vq is the tree-level effective potential, is the 
one-loop contributions at zero temperature. 

Vo, the tree-level effective potential, is given by 

W) = \{m% - c)cj? + ^ (Ai + j£) 4 . (31) 
one-loop contribution at zero temperature, is 

given by 



1 



64tt 2 



i—h ,77 ,7r ,q' 



In 



m 2 (0) 



(32) 



rrii (</>) and rii are the effective masses depending on <f> and 
the number of degrees of freedom, respectively, which are 
given in Table HI In the calculation of the loop integral 
in v/ ^ , we have taken the limit A — > 00 and have used 
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the MS scheme with a slight modification to renormalize 
the ultraviolet divergences. The first term is the one-loop 
contribution in ordinary MS scheme with the renormal- 
ization scale /i. The modification is the existence of the 
second term which are added to preserve the tree-level 

VEV 0o = V~ 2 ( m l ~ c )/( A i + ^l N f) and ' then > we 
set <f>Q = u(=246GeV). The parameter A is determined 



through = dV^ {<j)) /d(j) at <fi = v and is given by 



A=-- 



16tt 2 



niaim?(v) ( In 



m?(v) 



- 1 



(33) 



At one-loop, the Higgs boson mass rrih is shifted from 
the tree-level value (m/j) tree = ^/Ai + Xs/NfV. In this 
paper, we adopt the following definition for the Higgs 
boson mass rrih at one-loop: 



Ai 



N, 



v 

8^ 



E 

=h,(,q' 



n 7 a: 



In 



(34) 



This is the curvature of the effective potential at = v : 
y(°)"(0 = v) with neglecting the contributions from the 
light or massless scalar bosons rj and 7rs [135]. 

As for the mass of the extra scalar bosons £, rj and 
the fourth-family quarks q' , we adopt the formula at the 
tree-level: 



Kit = 2c - 



— v , m rj = 2c , m g , = — v . (35) 



In the following analysis, we use these definitions for the 
masses. 



where Jb and Jf are defined by 



■7b (a) 



da; a: 2 In ( 1 - e "V^+^ 



Jp(a) = y di In ( I ( 



(37) 



In the following analysis of the EWPT, we carry out a nu- 
merical integral for Jb and Jf without high temperature 
expansion. 

In the ordinary perturbation theory at finite temper- 
ature, the perturbative expansion breaks down near the 
critical temperature due to the existence of the higher- 
loop IR divergent diagrams in the massless limit. To 
improve the reliability of the perturbative expansion, we 
include the contributions from the ring diagrams which 
are the most dominant IR contributions at each order of 
the p erturbative expansion [7(| [73, [zl, [zl, US HH, [HJ 

One can include the contribution of ring diagrams, 
^ring(</>, T), by replacing m 2 ((/)) (i = h, £, 77, n) in Vi (0) and 
V x with the effective T-dependent masses AAf{(f),T) = 
mf((f>) + H<s>, where 11$ is the self-energy of the scalar 
bosons in the IR limit where the Matsubara frequency 
and the momentum of the external fields go to zero and 
in the leading order of nii{4>)/T and is given by 



B. Finite-temperature effective potential 



(T) 

The one-loop contribution at finite temperature, , 
is given by 



n$ = — [(N 2 f + l)Ai + 2N f \ 2 + N c y 2 }T 2 



(38) 



VP{^T) 
T 



^[ Y, ^JBlmU^/T^+n^JFlmK^/T 2 } 

i—h,^,r],7r 



at one-loop order. 



After all, the one-loop ring-improved effective potential 



(36) is given by 



V(<j>,T) =V W>) + V{ 0) (<P) + VP(^T) + y ing (0,T) 



=Fo(0) + ^ 2 + Y 



64tt 



rMtfaT) In 



M 2 MT) 



2tt 2 



Jb[MU^,T)/3 2 



(39) 



r 



In the following, we study the finite-temperature EWPT by numerically evaluating this effective potential. 
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IV. NUMERICAL ANALYSIS OF 
ELECTROWEAK PHASE TRANSITION 

The EWPT should be strongly first order in order 
to avoid the washout of the generated baryon asymme- 
try in the broken phase. How strongly first order the 
phase transition m ust be dep ends on the energy of the 
sphaleron solution [lOCt Il0l| in the model considered. 
As long as the classical (static) solution of the equation 
of motions is concerned, one may neglect the eff ect o f 
the fourth- family quarks even when they are heavy |l37j | . 
Then, one may use the condition 



0c/Tc=l.O, 0.8, 0.6, 0.4 (y=2.0, c=0) 



4>c/T c > 1, 



(40) 



as the criterion for a strongl y fi rst-order EWPT, as dis- 
cussed in our previous work [65(. 

In the following analysis, assuming a first-order phase 
transition, we solve the conditions 



dV{4>,T c ) 



0, V(tf> c ,T c ) = V(0,T o ), (41) 



numerically for various parameters, rrih, m^, m q ' and m n 
(see Eqs. (|34p. (pj)) for definitions). Then we evaluate 
4> C /T C in order to estimate the strength of the first-order 
phase transition. 

We first explore the parameter space of the renormal- 
izable theory without the constraint due to the compos- 
iteness condition, in order to locate the parameter re- 
gion where a strongly first-order EWPT is realized. We 
consider only the region where the stability condition at 
tree-level, Xi + X 2 /Nf > 0, is satisfied, otherwise the mass 
parameter squares in the one-loop effective potential be- 
come negative. We also consider only the region where 
the perturbation theory is reliable: 



Ai + \ 2 /N f < 



N f 



A2 (4ttV 



2 W 
v < - — — 



(42) 



In order to fulfill the requirement A > 1 TeV, we also 
concentrate on a region where y < 3.0. For an estimated 
A, we require that <f> = v is the global minimum of the 
one-loop zero-temperature effective potential V^°\<f) for 
< <fi < A. The renormalization scale of the effective 
potential is set to the electroweak scale, /i = v. We next 
examine the possible overlap of the parameter region of 
the strongly first-order EWPT and the region subject to 
the compositeness condition. 



a=i 




FIG. 6: Contour plots for various (j> c /T c in A1-A2/./V/ plane at 
y = 2.0, c = 0. The dashed lines are the contours of 4>c/T c = 
1.0,0.8,0.6,0.4 from bottom left to top right as indicated. 
The solid lines indicate the boundary of the allowed region 
with A > ITeV. (The region with 1 TeV> A > 0.5TeV is also 
shown.) In the shaded region, the effective potential at tree- 
level is unstable. The black filled-circle indicates the values 
of the couplings Ai , A2 /Nf when the compositeness condition 
is satisfied at Ajf= 9 TeV so that y — 2.0. 



(Sc/Tc=0.8, 0.6, 0.4 (y=2.5, c=0) 




FIG. 7: Contour plots for various (j> c /T c in A1-A2 /Nf plane 
at y = 2.5, c = 0. The dashed lines are the contours of 
4>c/T c — 0.8, 0.6, 0.4 from bottom left to top right as indicated. 
The black filled-circle indicates the values of the couplings 
Ai , A2 /Nf when the compositeness condition is satisfied at 
A 4 f= 2.3 TeV so that y = 2.5. 



A. Yukawa theory without compositeness condition 



We first discuss the numerical results for the Yukawa 
theory without the constraint due to the compositeness 
condition. In this analysis, we neglect the effect of the 
U(1)a symmetry breaking term, setting c = 0. 

In FIG. O we show the contours of 4> C /T C in the Ai~ 
Xi/Nf plane for y = 2.0 in the allowed region with A > 1 



TeV. (The contours with 1 TeV> A >0.5 TeV are also 
shown for reference.) We clearly see that the region of 
the strongly first-order transition lies above the stability 
boundary Xi + X 2 /Nf = 0. When Ai +X 2 /Nf gets larger, 
4> C /T C tends to become smaller. For a fixed small Ai + 
A2/-/V/, there is a tendency that <fi c /T c decreases as X 2 /Nf 



9 



increases. 

FIG. [7] is a similar contour diagram for y — 2.5. We 
notice that the region with 4> C /T C > 1 disappears in this 
case. The substantial region close to the stability bound- 
ary is excluded by the condition A > 1 TeV. This is 
because one encounters the instability at a lower scale 
for the larger Yukawa coupling. But, even when the al- 
lowed region is extended to A > 0.5 TeV, the transition 
is weakly first order with 4> C /T C < 1. For larger values of 
y, as far as we explored, the transition gets more weakly 
first order, or possibly second order. 

In FIG. [H we show the contours of <j> c /T c in y-Xz/Nf 
plane for Ai + X 2 /Nf = 0.05. We see clearly that a 
strongly first-order phase transition is realized even for 
y > 2.1, which corresponds to the experimental lower 
bound from the direct search of the fourth-family quarks 
m q' ^ 256 GeV, in the range 3 < Xi/Nf < 4 (corre- 
sponding to 430 GeV< <500 GeV). For larger values 
of y, however, the stability of the electroweak vacuum re- 
quires lager values of X 2 /Nf and then the strength of the 
first-order phase transition becomes weaker, or possibly 
second order. 

In FIG. we show the contours of the one- loop Higgs 
mass rrih in y-^/Nf plane for Ai + X 2 /Nf = 0.05, 
corresponding to {nih)trcc = 55 GeV. In this case, for 
M/Nf > 1 or y > 1 we can neglect the one-loop con- 
tributions of r\ and tts to rrih and the use of the formula 
Eq. (f34|) is validated. It is clear that for y > 2.1 the 
Higgs mass receives rather large one-loop correction and 
is heavier than experimental lower bound, rrih > 114GeV, 
in the whole region. 

In FIG. [lOl 4> C /T C is plotted as a function of rrih for 
several values of with my = 260 GeV fixed. We can 
see that with fixed, 4> C /T C decreases as rrih increases, 
while, with rrih fixed, 4> C /T C increases as toj increases. 
In the whole range, rrih exceeds the experimental bound, 



0c/Tc=2.5, 1.5, 1.0, 0.5, 0.3, 0.2 tU +A2/Nf=0.05, c=0) 




Am A 






A=1TeV 










A=0.5TeV : 
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(— m q =256GeV : 
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FIG. 8: Contour plots for various (f> c /T c in y-\2/Nf plane 
at Ai + X 2 /N f = 0.05, c = 0. The value of cj> c /T c for each 
contour is shown in the diagram. 
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FIG. 9: Contour plots for various mj, in y-\2/Nf plane at 
Ai + \-2/Nf — 0.05, c = 0. The dashed lines are the contours 
of m h = 114, 200, 300, 400, 500 GeV as indicated. 



m,-=260GeV, m,=0 



1.0 




150 200 250 300 350 400 

FIG. 10: 4> C /T C as a function of rrih for m q i = 260 GeV and 
m 5 = 400, 450, 500, 550 GeV from bottom left to top right. 



m { =450GeV, m,=0 




150 200 250 300 3S0 



FIG. 11: (j> c /T c as a function of rrih for — 450 GeV and 
m q , = 200, 260, 280, 300 GeV from top to bottom. 



m h > 114GeV. In FIG. [TTJ on the other hand, (j) c /T c is 
plotted as a function of rrih for several values of m q r with 
ni£ = 450GeV fixed. In this case, we note that with mh 
fixed, 4> C /T C decrease as m q > increases. 
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B. Effect of explicit symmetry breaking term 

We next examine the effect of the explicit U(1)a sym- 
metry breaking term by taking a value of m v (or c) non- 
zero. The effect can be read from FIG. Q21 For a fixed 
value of rrih, <t> c /T c decreases as m n increases. So we see 
that the explicit symmetry breaking term reduces the 
strength of the first-order phase transition. The experi- 
mental bound on the mass of the pseudo NG boson de- 
pends on how rj couples to the other particles, which is 
not specified in our model. If we adopt the bound for 
the pseudoscalar Higgs boson in supersymmetric models 
(with tan (3 > 0.4), the allowed values are m v > 93.4 GeV 
[l09( . For such a value of m ni w e see that the region of 
4>c/T c > 1 becomes quite narrow |l38j . 



m f =450GcV . m,, =260QcV 




0.2 I ' ' ' ' 1 ' ' ' ' 1 ' ' ' ' 1 ' ' ' ' 1 

150 200 250 300 350 

m,[G=V] 

FIG. 12: 4>c/T c as a function of mj, for = 450 GeV, 
m q , = 260 GeV and m v = 0, 50, 100, 150 GeV from top to 
bottom. 



Typical values of <j) c and T c are shown separately in 
FIG.QSlas a function of mh for m q t = 260 GeV, = 450 
GeV and m„ = 100 GeV fixed. 



m f =450GeV, m,=260GeV, m,,=100GeV 




180 200 220 240 260 280 300 
m,[GeV] 



FIG. 13: Plots of <f> c (black) and T c (gray) as a function of 
m h with m q , = 260 GeV, m e = 450 GeV and m v = 100 GeV 
fixed. 



C. Compositeness condition 

Finally, we discuss the numerical results for the 
Yukawa theory with the compositeness condition im- 
posed. Let us recall the plots in FIG. S] and FIG. [5] 



of the electroweak-scale values of the running coupling 
constants which are subject to the compositeness condi- 
tions Eqs. (|27l) (Criterion A) at various scales: the val- 
ues Ai, A2 for A4f = 1.0 TeV come close to the stabil- 
ity boundary at the electroweak scale, taking the value 
Ai + Ag/iV/ = 0.05 and it is located at the "cusp" on 
the boundary of the allowed region with A > 1 TeV. 
In this case, corresponding to the blue filled-circle (in- 
dicated with "A") in FIG. though we could not find 
the solution of Eqs. (|4"Tj) , we have checked that the phase 
transition is weakly first order with 4> C /T C < 0.001, or 
possibly second order. 

The above conclusion deserves discussions. In case if 
one adopts the other criterion for the compositeness con- 
dition, the phase transition can be first order. In the case 
of the criterion B, for example, the electroweak-scale val- 
ues of the running coupling constants which are subject 
to the compositeness condition at A 4 f = 3.7 TeV come 
close to the stability boundary at the electroweak scale, 
taking the value A x + A 2 /A/ = 0.05. In FIG. [51 we show 
the electroweak-scale values y, A 2 of this case by the red 
filled-circle (indicated with "B"). One immediately see 
that the phase transition is rather strongly first order 
with <fi c /T c ~ 1. Thus the critical behavior of the phase 
transition is rather sensitive to the choice of the values 
of the coupling constants at the compositeness scale A 4 f. 
However, combining with the results of the Yukawa the- 
ory without the compositeness condition, in particular, 
with the result for y = 2.5 shown in FIG. [71 it seems fair 
to say that the four-fermion interaction of the fourth- 
family quarks, which causes EWSB at zero-temperature 
and produces the mass of the heavy quarks larger than 
77v - 310 GeV (y> 2.5, A 4f < 2.3 TeV), does not lead 
to th e strongly first-order EWPT at finite temperature 
[l39| . We note that a region with A 4f < 1 TeV (y > 3.0, 
rriqi > 370 GeV) is beyond the scope of our analysis using 
the effective renormalizable theory. 




200 ■ — ■ — 1 — 1 ' — ' — 1 — 

200 500 1000 2000 5000 , x 10 4 

HGcV] 

FIG. 14: Plots of the running mass of the fourth-family 
quarks, fh q i(fi) = y(n)v/ a/2V/ , as a function of the renor- 
malization scale [i. The value of "on-shell" mass is defined 
by m q >, p hys = y( m q',ph y s)v/^/2Nf. The dot-dashed line (or- 
ange) indicates m q i (fi) — fj,. 



11 



V. CONCLUSION AND DISCUSSION 

In this paper, we have studied the finite-temperature 
electroweak phase transition in the model where the elec- 
troweak symmetry is dynamically broken due to the four- 
fcrmion interaction of the fourth-family quarks. Based on 
the effective renormalizable Yukawa theory with/without 
compositeness condition, we have estimated the strength 
of the first-order phase transition <fi c /T c , using the finite 
temperature effective potential at one-loop with the ring- 
improvement. In the Yukawa theory without composite- 
ness condition, the phase transition can be strongly first 
order with 4> C /T C > 1 for the experimentally acceptable 
Higgs boson and fourth-family quarks masses in the range 
256 GeV < m q > < 290 GeV when X 1 + X 2 /N f = 0.05. On 
the other hand, once the compositeness condition is im- 
posed as the boundary condition for the running coupling 
constants, the values of the couplings at the electroweak 
scale are restricted in a certain region of the parameter 
space. There, the phase transition turns out to be weakly 
first order, or possibly second order. The above result de- 
pends on how to specify the compositeness condition. In 
fact, we observed that the values of the running couplings 
at fx = v are rather sensitive to the choice of the bound- 
ary condition at fi — A 4 f and if one takes the smaller 
values as the boundary condition at fi — A 4 f (for exam- 
ple, Criterion B), the phase transition can be strongly 
first order with (j) c /T c ~ 0(1). In spite of this ambiguity, 
combining with the results of the Yukawa theory without 
the compositeness condition, it seems plausible that for 
3-0 > y > 2.5 (corresponding to 1 TeV < A 4f < 2.3 TeV 



and 480 GeV > m q > lPhya > 330 GeV "on shell" [140|), 
the four-fermion interaction of the fourth-family quarks, 
which causes EWSB at zero temperature, does not lead 
to the strongly first-order EWPT at finite temperature. 
We note that a region with y > 3.0 is beyond the scope 
of our analysis using the effective renormalizable theory. 

Since the critical behavior of first-order phase transi- 
tion at finite temperature that we concern, is not univer- 
sal in general, the result of our analysis would depend on 
our choice of low energy effective theory, where a c ertain 
truncation of fields and operators has to be done [X lOj ] - 
Our analysis, therefore, must be semi-quantitative (or 
qualitative) , showing a possibility to realize the strongly 
first-order EWPT required for the electroweak baryoge- 
nesis. 

Fortunately, it is now possible to formulate the model 
considered in this paper non-perturbatively on the lat- 
tice, preser ving SU(2)xU ( l) ch iral gauge symmetry ex- 
actly [m, Gil EM [Til, UM [HE lll3l- A direct nu- 
merical analysis of EWPT by Monte Carlo s i mulations 
with the state-of-art technique [ill EM EM EM EM 

EH, EM EM Ell ma y shed n g ht on the issues dis- 
cussed above. Even in the global symmetry limit, where 
SU(2)xU(l) gauge interactions are turned off, there are 
several questions worth studying: one may study the 
phase structure and the critical behavior of chiral sym- 
metry restoration at finite temperature in relation to the 



triviality bounds for the masses of th e Higgs bosons and 
the fourth- family quarks and leptons [l27l Il28l Il29j . 

In this paper, we have focused on the effect of the 
fourth- family quarks. The analysis of the effect of the 
fourth-family charged lepton and neutrino will be re- 
ported in a subsequent paper [9?| . 
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APPENDIX A: SCALAR POTENTIAL 

In this appendix, we give scalar potential in the full ef- 
fective Yukawa theory which includes both fourth-family 
quarks and leptons as well as all scalar fields H T i and 
X Q (« = 1,2,3). 

The most general renormalizable scalar potential 
which is consistent with symmetry (including the gen- 
eral U(1)a breaking terms) is given by 



V($,H T ,,x) =V 2 + V 3 + Vi 



(Al) 



where 



V 2 = m%tY(^^)+m 2 HT Hl,H Tl +m 2 x x a *X a 

-c(det$ + c.c.), (A2) 
V 3 = aH^eT a x a *H T ,+c.c, (A3) 



and 

r 4 



i,j,k,l 

+ a{Hl,H T .f + p 1 { X a *X a f + p2(x a X a )(.X b *X b l 



1,3 



+ Z>£ tr(<^-)x Q *X a + KHl,H T ,x a *X a (A4) 

where we have used the following notation 

01 = $ , <t> 2 = (A5) 

and i,j, k, I = 1, 2. 



APPENDIX B: RENORMALIZATION GROUP 
EVOLUTION OF THE FOUR-FERMION 
THEORY 

Just below the scale of the four-fermion interaction /.t < 
A4f, the four-fermion theory equivalent, Eq. ([3"|h 

C' i{ = q'iQq 1 - yMQ'L^QR + «•) - ™| tr(*^) (Bl) 
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is renormalized as Then, the renormalized couplings read 

Cq = q'ifiq' - y q >o{q' L $q' R + c.c.) 

+ Z«tr(fl„$ t fl' i $)-n4tr$t$ (B2) 

_ ^( t r*t$)2_^ tr( $t$ ) 2 +c(det$ + c . c .) ig ^ 2 i 

in terms of the bare field variables, where 



Tl% = "1*0 ~ lg^2^'o( A 4f - M )> ( B4 ) 
9 /V 

A 2 = ^y^M^/A (B5) 

Ai = 0, c = 0. (B6) 



^ - I^oM^/A (B3) w = ^! rTT l TTT . (B8 ) 



XiQi) = 0, (B7) 



N c ln(A 2 f /^2 
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The model of dynamical electroweak symmetry break- 
ing due to top quark condensate has been proposed by 
Nambu and Miransky-Tanabashi-Yamawaki [33 . l40l lilfl 
and examined in detail through renormalization group 
methods by Marciano and Bardeen-Hill-Lindner [42i. 143. 
0. 

The original Higgs sector of the SM, if the electroweak 
interactions is turned off, is nothing but the 0(4) linear 
sigma model and its finite temperature phase transition 
is second order, which is governed by the Wilson-Fisher 
IR-stable fixed point. It is the effect of the gauge in- 
teraction which makes the fixed point IR-unstable and 
causes a first-order phase transition [5l| ] . Once the Higgs 



Kallarackal, 



and 



sector is extended, the number of the scalar fields is in- 
creased and there appear additional quartic couplings 
among them. The fixed points of the multiple quartic 
coupling constants may be IR-unstable and one can ex- 
pect a first-order phase transition even in the pure scalar 
sector [52l [56| . A related approach is to consider the 
single Higgs doublet model (the O (4) model) with the 
dimension-six or higher operators [60l . l6ll . [63 . HH, |64| . 
The quartic coupling may be then assumed negative 
so that the model is out of the domain of the Wilson- 
Fisher fixed point. Such higher dimensional operators 
may be induced by the effect of heavy particles coupled 
to the Higgs doublet, or more generally, by the effect 
of a certain dynamical system behind the Higgs sector. 
The EWPT has been examined in various dynamical 
models of the Higgs sector: walking technicolor theories 
in [65l . I06I . [67| using low-energy effective sigma models, 
pseudo Goldstone Higgs boson models or little Higgs 
models in 1641. 6811 . models of the gauge-Higgs unifica- 
tion in [II Eg, El- 

[133] V includes a scalar cubic term H^, er a x a * H T i which may 
enhance a strength of a first-order phase transition at 
high temperature. 

[134] If SU(3)xSU(2)xU(l) gauge interactions are included, 
there appears an effective infra-red fixed point in the 
renormalization group equation of the Yukawa coupling 
Eq. JT6J) HHH. But its value is about y c ~ 1.6, which 
is less than the values of y considered in our analysis. 
So we neglect the effect of the infra-red fixed point. 

[135] The contributions of rj and 7rs are divergent and may be 
regarded as artifacts due to the fact that V^°'"((f> = v) 
corresponds to the off-shell Higgs boson mass at the zero 
momentum p 2 — 0, while the on-shell (physical) mass 
should be finite. To neglect the contributions from 77 
and 7rs are expected to be valid as long as Ai + 
is much smaller than X 2 /Nf or y. This argument is in 
line with |75j . 

[136] The ring-improved perturbation theory is valid when 
the non-ring diagrams are suppressed with respect to 
the ring-diagrams. We infer the expansion parameter 
in the ring-improved perturbation theory by the power 
counting argument as in the \<f> 4 theory [7g]. By in- 
specting the higher order diagrams for the scalar field 
self-energies, the expansion parameters are expected 
to be m max(Ai + X 2 /N f , X 2 /N f )T /[4wMi{<f>,T)] (i = 
ft, £,77, 7r), where Mi(<j),T) and n; are the effective T- 
dependent masses of the scalar bosons and the corre- 
sponding numbers of degrees of freedom. This is max- 
imized for i — 7r and (f> — 0, then, (Nj — l)max(Ai + 
M/Nf, A 2 /A r /)T c /[4 7 r.M vr (0,T c ) < 1 should be held for 
the ring-improved perturbation theory to be valid at 
T = T c . In the analysis of the EWPT, we have ob- 
served that for y > 2.1 the above expansion parameter 
is smaller than 1 but is not so small (greater than 0.5). 
[137] The large Yukawa couplings of the fourth-family quarks 
and leptons may affect the baryon number diffusion rate, 
which is usually computed at one-loop including the ef- 
fect of the fluctuations around t he sphaleron solution 
[lOl , [Tol . Hoi . Hol flOrj . [l07l . [l0^ ]. 

We note that the tree-level formula is used for ra, in this 
analysis and an inclusion of the one-loop correction to 



[138] 

[139] There remains a possibility that the effect of the elec- 



would affect the extent of the region with <f> c /T c > 1. 
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troweak interactions might enhance slightly the strength 
of the first-order phase transition. 
[140] For y = 3.0 ( m q , = 370 GeV), the fourth-family quark 
mass defined by Eq. (|35[) receive rather large correction 
due to the running of the Yukawa coupling. The "on- 



shell" mass, defined by m q ,^ hya = y(m q , }Phya )v/ v / 2N f , 
is m q i }Phys ~ 480 GeV for y — 3.0 and is m q i }Phya ~ 330 
GeV for y = 2.5 (See FIG. EJ. 



